Virtual environments are ubiquitous in computing environments, finding use in video games (in which a virtual environment may represent a game world); maps (in which a virtual environment may represent terrain to be navigated); simulations (in which a virtual environment may simulate a real environment); digital storytelling (in which virtual characters may interact with each other in a virtual environment); and many other applications. Modern computer users are generally comfortable perceiving, and interacting with, virtual environments. However, users' experiences with virtual environments can be limited by the technology for presenting virtual environments. For example, conventional displays (e.g., 2D display screens) and audio systems (e.g., fixed speakers) may be unable to realize a virtual environment in ways that create a compelling, realistic, and immersive experience.
Virtual reality (“VR”), augmented reality (“AR”), mixed reality (“MR”), and related technologies (collectively, “XR”) share an ability to present, to a user of an XR system, sensory information corresponding to a virtual environment represented by data in a computer system. Such systems can offer a uniquely heightened sense of immersion and realism by combining virtual visual and audio cues with real sights and sounds. Accordingly, it can be desirable to present digital sounds to a user of an XR system in such a way that the sounds seem to be occurring—naturally, and consistently with the user's expectations of the sound—in the user's real environment. Generally speaking, users expect that virtual sounds will take on the acoustic properties of the real environment in which they are heard. For instance, a user of an XR system in a large concert hall will expect the virtual sounds of the XR system to have large, cavernous sonic qualities; conversely, a user in a small apartment will expect the sounds to be more dampened, close, and immediate.
Existing technologies often fall short of these expectations, such as by presenting virtual audio that does not take into account a user's surroundings, leading to feelings of inauthenticity that can compromise the user experience. Observations of users of XR systems indicate that while users may be relatively forgiving of visual mismatches between virtual content and a real environment (e.g., inconsistencies in lighting); users may be more sensitive to auditory mismatches. Our own auditory experiences, refined continuously throughout our lives, can make us acutely aware of how our physical environments affect the sounds we hear; and we can be hyper-aware of sounds that are inconsistent with those expectations. With XR systems, such inconsistencies can be jarring, and can turn an immersive and compelling experience into a gimmicky, imitative one. In extreme examples, auditory inconsistencies can cause motion sickness and other ill effects as the inner ear is unable to reconcile auditory stimuli with their corresponding visual cues.
The present invention is directed to addressing these shortcomings by presenting a virtual sound to a user, with the presentation of the sound incorporating one or more playback parameters based on aspects of the user's real environment. For example, the presentation can incorporate a simulated reverb effect, where one or more parameters of the reverb depend on attributes of the user's real environment, such as the cubic volume of the room, or the materials of the walls of the room. By taking into account the characteristics of the user's physical environment, the systems and methods described herein can simulate what would be heard by a user if the virtual sound were a real sound, generated naturally in that environment. By presenting virtual sounds in a manner that is faithful to the way sounds behave in the real world, the user may experience a heightened sense of connectedness to the mixed reality environment. Similarly, by presenting location-aware virtual content that responds to the user's movements and environment, the content becomes more subjective, interactive, and real—for example, the user's experience at Point A can be entirely different from his or her experience at Point B. This enhanced realism and interactivity can provide a foundation for new applications of mixed reality, such as those that use spatially-aware audio to enable novel forms of gameplay, social features, or interactive behaviors.